Knowledge-Acquisition Interfaces for Domain Experts: An Empirical Evaluation of Protégé-2000

Application experts need to be able to maintain, populate, and verify existing knowledge bases in order to use knowledge-based tools to perform their daily tasks. Protégé-2000 is a tool that enables experts to perform the knowledge-acquisition and maintenance process. In this paper, we describe a controlled study that we performed to measure both the efficiency and quality of knowledge acquisition by domain experts and to evaluate how well the domain experts can retain their proficiency in using a knowledge-acquisition tool. We adopted the techniques that are used to evaluate usability of traditional software products to evaluating knowledge-acquisition process. Using Protégé-2000 and its domain-specific extensions, the domain experts with no previous experience with knowledge-based systems were able to perform complex knowledge-acquisition tasks efficiently and correctly. Our data show that subject-matter experts were able to use both tools after only 1-2 hours of training, and entered knowledge correctly overall 94% of the time. The custom-developed user interface greatly enhanced the ability of experts to identify known errors in a knowledge base (93% vs. 48% detection). The experts also were able to retain their skills over 1-2 week period of time without using the system. 2 " Studying AI systems is not very different from studying moderately intelligent animals such as rats " The use of knowledge-acquisition systems usually has been the realm of trained knowledge engineers or at least users with substantial computer experience. However, the ontologies and knowledge bases that drive modern intelligent systems need to be maintained. Menzies (Menzies 1999) argues that " we must move the focus of knowledge engineering from knowledge acquisition to knowledge maintenance. " If participation of system analysts at the stage of knowledge-base development is feasible (if not always desirable), domain experts themselves must perform knowledge-base maintenance, quality control, and update. Professionals want to be responsible for the knowledge-based systems used in their application area. However, most of the tools for knowledge-base development and maintenance, which exist today, are geared toward not just the extremely computer-literate user, but toward the computer-science-educated user (Duineveld et al. 1999). In order to allow domain experts to use knowledge-base construction and maintenance tools (hereafter referred to as knowledge-base editors) the developers of these tools need to make the tools easier to use. The impediments to developing user-friendly knowledge-base editors are enormous: not only are domain experts not used to thinking in terms of ontological categories but also, in most …